While Reinforcement Learning (RL) has been proven essential for tuning large language models (LLMs), it can lead to reward over-optimization (ROO). Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. Additionally, KL regularization focuses solely on regularizing the language policy, neglecting a potential source of regularization: the reward function itself. Inspired by demonstration-guided RL, we here introduce the Reward Calibration from Demonstration (RCfD), which leverages human demonstrations and a reward model to recalibrate the reward objective. Formally, given a prompt, the RCfD objective minimizes the distance between the demonstrations' and LLM's rewards rather than directly maximizing the reward function. This objective shift avoids incentivizing the LLM to exploit the reward model and promotes more natural and diverse language generation. We show the effectiveness of RCfD on three language tasks, which achieves comparable performance to carefully tuned baselines while mitigating ROO.

From Zac Efron to Margot Robbie, many of the world's most beautiful celebrities are known for their square faces. Now, a new study claims that people with this face shape are seen as more aggressive than those with oval faces, such as Rihanna and Ben Affleck. Researchers from the University of New South Wales measured the facial-width-to-height ratio (FWHR) of 17,607 passport images of male and female faces, before asking people to rate them for aggression. The results revealed that faces with a high FWHR (square faces) were rated as more aggressive than people with low FWHR (oval faces) – particularly if they belonged to young men. From Zac Efron to Margot Robbie, many of the world's most beautiful celebrities are known for their square faces Researchers from the University of New South Wales measured the facial-width-to-height ratio (FWHR) of 17,607 passport images of male and female faces, before asking people to rate them for aggression.

How to Do It is Slate's sex advice column. Send it to Stoya and Rich here. My first crush ever was on my uncle. I've noticed an attraction to two of my cousins. I've never, ever considered acting on these desires or told anyone, but I'm wondering if this is normal. Is my brain missing the evolutionary programming that makes you not want to fuck your family?

How to Do It is Slate's sex advice column. Send it to Stoya and Rich here. I have been dating my boyfriend for about 10 months now. I'm struggling with information he shared with me very early on. At the beginning of our relationship, he told me that he has seen escorts before.

How to Do It is Slate's sex advice column. Send your questions for Stoya and Rich to howtodoit@slate.com. I have been sexually active since I was 17. I am now 29 years old. A majority of the sex I had between 17 and 21 was only when I was drunk, so I don't remember most of it, but I know I didn't climax.

A Washington state man, who survived a samurai sword attack last week by his now ex-girlfriend while he was asleep, revealed his rigorous video game training regime left him with a lack of sex drive, and this resulted in his former partner thinking he was cheating on her. "It killed my sex drive. I was training too hard, it exhausted me. I felt bad because she needed the affection. I just couldn't keep up," 29-year-old Alex Lovell told Buzzfeed News of his video game training regimen, which led to the attack by his ex-girlfriend March 3. "She thought I was having sex with other people," Lovell added.

If we are to believe research conducted over the last decade or so, the answer is: a lot. And according to a recent study, facial dimensions can even reveal a person's sex drive. While men had a higher sex drive than women, in both sexes it was people with relatively wider faces that had this higher sex drive. On its own, this finding seems surprising at best and crazy at worst – how could the shape of someone's face possibly have anything to do with their libido? When viewed in the context of other studies, however, it starts becoming clearer that there may be some logic and truth to it.